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Conmiissioner for Patents: 

I, David V. Carlson, declare and state that: 

I am an attorney of record in this application and admitted to practice before the 
Patent and Trademark Office, Registration No. 31,153. 

I, David V. Carlson, declare and state that: 

Attached hereto as Exhibit A is a true copy of the Express Mail Post Office 
Receipt, customer copy. The customer copy of the Express Mail shows a received date of 
August 12, 2003, at the U.S. Postal Service. It also bears a handwritten check-in date of August 
12, 2003, at 10:05 p.m. and has the initials of the acceptance clerk stated thereon. In addition, 
the fill! postage is noted as being paid. This is Express Mail label EV336610850US. It is 
addressed to the Commissioner for Patents in Alexandria, Virginia. Enclosed as Exhibit B is a 
copy of the Utility Patent Application transmittal bearing my signature and a mailing date of 
August 12, 2003. Attached as Exhibit C is the application data sheet bearing in the upper right- 



hand comer the same Express Mail number and stating the customer number, the title, and the 
inventors' names. Enclosed as Exhibit D is a copy of the utility patent application, with figures 
bearing in the top portion the same Express Mail label number EV336610850US as evidence that 
this is what was mailed on August 12, 2003. 

Enclosed herewith as Exhibit E is the postcard which has stated thereon the 
Express Mail label together with the listing of the documents that are enclosed therewith. 

Attached as Exhibit F is a photocopy of an Express Mail package received from 
the Post Office on August 22, 2003. In particular, the U.S. Postal Service returned to the law 
firm of Seed Intellectual Property Law Group the Express Mail package which had been 
entrusted to their care on August 12, 2003. The package was returned with the contents present 
but some comers and sides had been partially ripped and the envelope was extremely wet and 
dirty. It appears to have been dropped in a puddle outside or some other very wet environment. 
Accordingly, the address and information on the face of the label had been substantially 
obliterated by the rough handling by the United States Post Office such that it was retumed to the 
sender. 

Inmiediately upon receipt of the package on August 22, 2003, appUcants prepared 
a second application and quickly filed a follow-up application on August 25, 2003, which 
contained a refiling of the same patent application. 

The Express Mail package which was retumed in damaged condition has still not 
been opened, but it could be provided in the same form to the United States Patent and 
Trademark Office if they wish to receive it in its partially tom and still unopened condition. 

Applicants formally request that the filing date of August 12, 2003, be accorded 
to the present application. 

I fiirther declare that all statements made herein of my own knowledge are true 
and that all statements made on information and belief are believed to be true; and further, that 
these statements were made with the knowledge that the making of willfully false statements and 
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the like is punishable by fine or imprisonmmt, or both, under Section 1001 of Title 18 of the 
United States Code, and may jeopardize the validity of any patent issuing fi"om this patent 
application. 

Dated this 17* day of December, 2003. 

David V. Carlson 
Registration No. 3 1 , 1 53 
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EXPRE MAIL NO. EV336610850US 



METHOD FOR FILTERING THE NOISE OF A DIGITAL IMAGE SEQUENCE 

BACKGROUND OF THE INVENTION 

Field of the Invention 

The present invention concerns the processing of digital images and, 
5 more particularly, a method for filtering noise in a digital image sequence. 

Description of the Related Art 

Digital images are currently being used in numerous applications, 
including those related with such traditional acquisition devices as still and video 
cameras. It is to be expected that ever greater use of digital images will be made 
10 in such new generation devices as mobile multimedia communication terminals. 

There exist numerous devices or applications that use digital images 
In sequence, that is to say, images acquired one after the other, separated by a 
brief interval of time and representing approximately the same real scene. 

The speed with which the sequence is acquired, i.e., the number of 
15 images acquired in a given time interval, may vary according to the specific 

application; for example, this number is very large in digital video cameras (about 
25 images per second) and smaller (about 15 images per second) in mobile 
communication terminals, which acquire the digital images and then transmit them 
in real time to a remote terminal. 
20 It is well known that digital image acquisition devices, especially 

when they include CMOS sensors, will intrinsically introduce noise into the 
acquired images. 

In digital image sequences noise not only degrades the quality of the 
images, but also reduces the encoding/compression efficiency. Indeed, the 
25 acquired image sequences have commonly to be encoded/compressed by means 
of encoding/compression techniques that operate in accordance with, for example, 
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the MPEG standard or the H263 standard and nowadays are very widely used in 
the greater part of devices in the market today. 

The encoding/compression efficiency becomes reduced by the 
presence of noise, because the introduced noise is typically in the form of random 
5 fluctuations that reduce redundancy both within an image and between images 
that are temporally close to each other. 

There exist numerous filtering techniques intended to reduce or 
eliminate the noise present in an image sequence. 

Numerous attempts have been made to develop efficient techniques 
10 for reducing the noise of a sequence by using various specific types of filters. 
Known digital filters include, for example, low-pass filters, median filters, adaptive 
spatial filters and recursive temporal filters with or without motion compensation. 

Other prior art techniques seek to improve noise reduction efficiency 
in image sequences by having recourse to hybrid methods that combine digital 
15 spatial filtering with digital temporal filtering. * 

Though the known techniques for reducing noise in image 
sequences are satisfactory in many respects, they are also associated with 
riumerbus drawbacks and problems that are bound up with, for example, 
inadequate performance, processing complexity and excessive processing costs 
20 that make it difficult to employ them in portable acquisition devices of a commercial 
type. 

BRIEF SUMMARY OF THE INVENTION 

The present invention therefore sets out to make available a method 
for reducing noise in an image sequence. This aim is attained with a method for 
25 filtering a sequence of digital images in CFA fomiat as described in Claims 1 to 12 
attached hereto. 

Another object of the present invention is to provide a filter as 
described in Claim 13 and an acquisition device as described in Claim 14. 
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BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS 

Further characteristics of the invention and the advantages 
associated therewith will be more readily understood from the detailed description 
about to be given of a preferred embodiment thereof, which is to be considered as 
5 an example and not limitative in any way, said description making reference to the 
attached drawings of which: 

Figure 1 shows the block diagram illustrating a possible acquisition 
device that implements a method in accordance with the present invention; 

Figure 2 shows the pattern of the filtering elements of a Bayer sensor 
10 that can be used in the device of Figure 1 ; 

Figure 3 schematically illustrates the succession of phases of a 
method in accordance with the present invention; 

Figure 4 shows a selection mask for selecting green-colored pixiels 
that can be employed in the method in accordance with the present invention; 
15 Figure 5 shows two selection masks for selecting red-colored pixels 

and blue-colored pixels that can be employed in the method in accordance with the 
present invention; 

Figure 6 shows one of the phases of the succession illustrated by 
Figure 3 in greater detail; 
20 Figure 7 shows an example of selecting pixels in accordance with a 

DRT selection; 

Figure 8 is a graph that illustrates the experimental results; and 
Figure 9 is a schematic illustration of two filtering architectures, the 
first in accordance with the present invention, the second of a conventional type. 

25 DETAILED DESCRIPTION OF THE INVENTION 

The preferred embodiment example of the present invention relates 
to a portable device capable of acquiring digital image sequences for video 
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applications and, more particularly, concerns the noise filtering of an image 
sequence acquired with a digital video camera. 

In this connection it should be noted that the teachings of the present 
invention can be extended also to applications other than those to which explicit 
5 reference is made in the description about to be given, for example, to the 

acquisition of image sequences in mobile multimedia communication terminals of 
the new generation. 

Figure 1 provides a very schematic illustration of a digital video 
camera 1 in the form of function blocks. The video camera 1 includes an 
10 acquisition block 2 that comprises an optical sensor 3. 

The optical sensor 3, which may be - for example - of the CCD 
(Charge Coupled Device) or the CMOS (Complementary Metal Oxide 
Semiconductor) type, is an integrated circuit comprising a matrix of photosensitive 
cells, each of which serves to generate an electrical signal proportional to the 
15 quantity of light that strikes it during the acquisition interval. Each photosensitive 
cell of the sensor, which is commonly referred to by the term pixel, corresponds to 
a respective pixel of the digital image. 

In a preferred embodiment the sensor 3 comprises an optical CFA 
(Color Filter Array) filter, for example, with a Bayer-type matrix. 
20 As is well known to persons skilled in the art, in a sensor with a CFA 

filter only a single photosensitive cell is available for acquiring a pixel. The sensor 
is covered by an optical filter constituted by a matrix (a Bayer matrix, for example) 
of filtering elements, each of which is associated with a photosensitive cell. Each 
filtering element transmits to the photosensitive cell associated with it the light 
25 radiation corresponding to the wavelength of only red light, only green light or only 
blue light, so that for each pixel it detects only one component (of which it absorbs 
no more than a minimal part). The pattern of the filtering elements in a Bayer filter 
is shown in Figure 2, where the letters R.G.B indicate, respectively, the red, green 
and blue elements. 
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The video camera 1 also includes an analog/digital (A/D) conversion 
block, indicated by the reference number 4. to translate the generated electric 
signal into a digital value with a predetermined number of bits (generally 8, 10 or 
12 bits). One may assume, solely by way of example and without thereby 
5 introducing any limitation whatsoever, that in the present invention the A/D 

converter 4 is such as to encode the incoming analog signals with eight-bit digital 
values. 

On the output side of the A/D block 4 the digital image Is in a video 
format, for example, it may be in a CFA (Color Filter An-ay) format, since each pixel 

10 is constituted by just a single chromatic component (R, G or B). For this reason, a 
single one-byte digital value is associated with each pixel. In one embodiment, the 
digital image may be in the CFA fomiat, but in other embodiments, other formats 
rriay be used and these are included within the concept of the invention. Thus, the 
reference to CFA herein should be understood to be one example of how to carry 

15 out the invention. 

A filtering block 5 - in this example of the Bayer type - is such as to 
filter the noise by operating directly on the digital CFA images of the sequence, 
producing for each noisy CFA image on its input side a CFA image with reduced 
noise on its output side. 

20 A pre-processing (PrePro) block 6, active before and during the 

entire acquisition phase, is such as to interact with the acquisition block 2 and to 
extract from the CFA image a number of parameters useful for carrying out 
automatic control functions: self-focusing, automatic exposure, correction of sensor 
defects and white balancing. 

25 A block 7, the IGP (Image Generation Pipeline) block, is designed to 

perfonn a processing phase that, starting from the digital CFA image, will produce 
a complete digital image - YCrCb format, for example - in which each pixel will 
have associated with it three digital values {i.e., a total of 24 bits) corresponding to 
a luminance component Y and two chrominance components Cr and Cb. This 
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transformation, known by the name of color interpolation, involves a passage from 
a representation of the image in a single plane (Bayer plane), which nevertheless 
contains infomiation relating to different chromatic components, to a 
representation in three planes. 
5 In digital still cameras the IGP block is commonly realized in the form 

of a dedicated processor. In one embodiment, this is a CFA processor, which may 
be implemented in VLSI (Very Large Scale Integration) technology. 

Preferably, the IGP block 7 in this example is also such as to 
perform, over and above the interpolation, various other functions, including - for 
10 example - the application of special effects, gamma correction, scaling, 
stabilization and other functions that will generally vary from one producer to 
another. 

This is followed by a compression/encoding block 8, which in this 
example is of the MPEG type (but could also be of other types. H263 for example). 
1 5 and a memory unit 9. 

When shooting a video sequence with the video camera 1, the 
sequence images are acquired consecutively by means of the acquisition block 2. 
preferably separated only by a brief time interval between one imagie and the next. 
The MPEG-4 standard, for example, requires fifteen images to be acquired per 
20 second. 

Hereinafter we shall use lmgi, Img2, Imga, ImgnW. Imgn, lmgn+i,... 
to indicate the images acquired in sequence: lmgi represents the first image of the 
sequence to be acquired. Imga represents the second image, and so on. 

Following acquisition, each image is processed by the subsequent 
25 blocks, so that in all the subsequent processing phases the images will still be 
processed in the temporal order in which they were acquired. 

Once they have been acquired, the sequence images are converted 
into digital values by the A/D converter 4. 
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the CFA format digital images aris then sent as input to the noise 
filter block 5 (CFA NF) to be processed in accordance with the noise filtering 
method of the present invention. As output the filter block produces a sequence of 
filtered CFA images, respectively, fjmgi, fjmga, fjmga, fjmgn-i. fjmgn. 
5 f Jmgn+1, each of which has less noise than on the input side. 

The filtered CFA images are then processed by the pre-processing 

blocks. 

On leaving the pre-processing block 6, each CFA image is sent to 
the IGP block 7. In this block the images are subjected to a color interpolation 
10 phase and therefore transformed into complete images, for example, in YCrCb 
format. 

The color interpolation phase may be performed, among others, by 

means of methods that are known to a person skilled in the art and are therefore 

obvious from the previous description. 
1 5 Thereafter the images are sent to the MPEG encoder block 8, which 

produces as its output a sequence or stream of images encoded/compressed in 

accordance with an MPEG encoding. 

The MPEG stream of compressed images may be recorded in a 

memory unit 9 or sent to an external peripheral device not shown in the figure. 
20 In a preferred embodiment the processing method filters the 

sequence of CFA images one at a time, in this example by means of a Bayer filter. 

The CFA images are filtered pixel by pixel, the scanning order being 

such that the pixels are systematically scanned from left to right and from top to 

bottom. In particular, for each pixel Pn(x,y) of an image Imgn there is calculated a 
25 respective filtered homologous pixel fj3n(x,y) of a corresponding filtered image 

fJmgn. 

When processing the pixel Pn(x,y) of the image Imgn and calculating 
the filtered pixej f j)n(x,y), the method of the present invention makes 
advantageous use also of the filtered pixels forming part of an image f Jmgn.i of 
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the previously filtered sequence. More particularly, it utilizes the image f_lmgn.i 
obtained by filtering the image lmgn.i that in the sequence temporally precedes the 
Image to be filtered imgn- 

Typically, therefore, three image buffers will be sufficient for carrying 
5 out the filtering process: two input buffers that contain, respectively, the image 
Imgn to be filtered (current image) and the previously filtered image fjmgn-i, as 
well as an output buffer containing the filtered current Image f_lmgn. 

Figure 3 shows a schematic representation of the succession of 
pheises of a processing method 20 for reducing noise In accordance with the 
10 present invention. 

In particular, Figure 3 shows the phases by means of which, starting 
from the pixel Pn(x,y) of the image Imgn. there is obtained the respective 
homologous filtered pixel f_Pn(x,y) of the con-esporiding filtered image f_lmgn. 

Given the Input pixel pn(x,y) to be filtered, a first selection phase 21 
1 5 (SW_sel) selects a first pixel set SWn(x,y) comprising the said pixel Pn(x,y) and a 
plurality of pixels forming part of the image Imgn in the neighborhood of said pixel. 
In one embodiment, the phase selects adjacent pixels that have associated with 
. them the same color (R, G or B) as the pixel to be filtered. 

In a preferred embodiment, the selection Is perfomied by using 
20 selection masks (or matrices) SM_G, SM_R, SM_B that differ on the basis of the 
color of the pixel pn(x,y) to be filtered, but all of dimension 5x5, for example, like 
those shown in Figures 4 and 5. 

Figure 4 shows a selection mask SM_G for the green pixels (G) in 
accordance with a particular embodiment of the invention. In the selection phase 
25 21 the mask SM_G Is aligned with the Image to be filtered In such a way that that 
Go Corresponds to the green pixel Pn(x,y) to be filtered. In this way the mask 
SM_G will select a first set of pixels SWn(x,y) comprising the green pixel pn(x,y) 
corresponding to Go and eight adjacent pixels situated, respectively, in positions 
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corresponding to the pixels Gi,...,G8 of the mask SM_G shown in the figure. This 
set defines a spatial working window SWn(x,y) for the green pixel to be filtered. 

Analogously, Figure 5 shows the selection masks SM_R and SM_B 
to be used, respectively, when the pixels to be filtered are red or blue, 
5 The mask SM_R for the red pixels is such as to select a first set of 

pixels SWn(x,y) comprising the red pixel Pn(x.y) to be filtered corresponding to Ro 
and eight adjacent red pixels situated, respectively, in positions corresponding to 
the pixels Ri , . . . .Rs of the mask. 

It should be noted that in this particular embodiment the selection 
10 mask SM_B for the blue pixels is identical with the selection mask SM_R for the 
red pixels. This choice is possible thanks to the particular pattern in which the 
filtering elements are arranged in a Bayer-type sensor 

This brings with It the advantage that the selection phase 21 
(SW_sel) has to discriminate only between two possible cases, namely to 
i5 distinguish whether the pixel Pn(x,y) to be filtered is or is not green. 

Coming back to Figure 3. a second selection phase 22 (TW_sel) 
selects a second set of pixels TWn(x,y), comprising pixels forming part of the 
previously filtered Image fjmgn-i aind arranged in corresponding positions, /.e., 
homologous with the pixels of the first TWn(x,y). 
20 to this end it will be advantageous to use the selection matrices 

described above, but this time applied to the previously filtered image fjmgn-i- 

The pixel set obtained in this manner defines a temporal working 
window TWn(x,y) for the pixel to be filtered. 

The temporal and spatial working windows represent the set of pixels 
25 that will play a part in the subsequent phases of the filtering process of the pixel 
Pn(x,y). 

As is well known to a person skilled in the art, when digital image 
sequences are filtered, a filtered pixel can be obtained by appropriately combining 
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a certain number of pixels that are adjacent to it either in space (spatial filtering), in 
time (temporal filtering) or in space/time (spatio-temporal filtering). 

In particuiar, as will be described in greater detail later on, the 
method of the present invention decides pixel by pixel whether the filtering to be 
5 used is to be exclusively spatial or, on the other hand, spatio-temporal. 

Advantageously, the decision regarding the type of filtering to be employed will be 
bound up with the amount of motion between successive images of the sequence, 
since this will make it possible to avoid motion compensation, a computationally 
very costly operation. 

10 When spatial filtering is employed, the operation will involve only the 

pixels of the spatial working window SWn(x,y), otherwise use will be made of the 
pixels forming part of both the windows. 

Once the two working windows - respectively in space and time - 
have been obtained, a first noise estimation phase 23 (Snoise_est) has as its first 

1 6 step tiie making of an estimate of a statistical parameter NLn(x,y) representative of 
the noise level present on the pixel Pn(x,y) and tiie respective spatial working 
window SWn(x,y). Henceforth we shall refer to this noise as spatial noise, while 
the phase will be referred to as spatial noise estimation. 

In greater detiail, the first step of the estimating phase 23 

20 (Snoise_est) is to make a preliminary noise estimate {i.e., to estimate a statistical 
parameter representative of the noise level) by means of a local calculation, that is 
to say, calculated for the pixels of the spatial working window SWn(x,y). As second 
step it obtains the definitive spatial estimate NLn(x,y) by modifying the preliminary 
estimate on the basis of a spatial estimate of the noise specific for the color of the 

25 pilxel to be filtered and specific also for the image Imgn. 

In greater detail, again, the spatial noise estimate is obtained by 
meanis of a computation of the recursive type that is made by taking into account 
not only a preliminary and local noise calculation, but also the spatial estimate of 
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the noise level miade for the last filtered pixel of the image Imgn having the same 
color as the pixel to be filtered Pn(x,y). 

Stated in mathematical terms, in the case in which, for example, 
Pn(x,y) is a green pixel, we have: 

NL„ix,y)^NL'i(x,y) = k„(x,y)xN[SW„ix,y)] + il-k„ix,y))xNt'Sp^ (1) 

where the superscript "G" indicates that the term relates to the color green, 
NlSW„(x,y)\ is the preliminary estimate calculated for the spatial working window 
SWn(x,y), k„ix,y) is a multiplication factor comprised between zero and one and 
detennines the strength of the spatial filter, NL^^pp^) is the spatial noise estimate 
made for the green pixel pp^ of the image Imgn that immediately precedes the 
green pixel to be filtered Pn(x,y) in the order in which the image Imgn is scanned. 

Obviously, if the pixel to be filtered Pn(x,y) is the first pixel of the 
riespective color to be filtered in the image Imgn, only the preliminary estimate 
N[SW„{x,y)\ will be available. In that case we can either put, for example, 
k,{x,y)=^ solely for the pixel Pn(x,y) or, alternatively, assign an arbitrary and 
preferably small value to the quantity NL^„(pp^). 

The ifirst phase of the spatial noise estimation 23 (Snoisejest) may 
be carried out, for exampje, as described In detail in European Patent Application 
No. 01830562.3 filed in the name of the present applicant, which is to be deemed 
to be wholly incorporated herein by reference. The meaning of the quantities 
NLn(x.y). N[SfV„ix,y)]. k„ix,y). NL^ipp^) and the manner in which they are 
calculated are likewise explained in that document. In this connection please refer 
to formulas (1). (2). (3). (4). (5), (6), (7), (8) and Figures 7, 8A, 8B, 10 (and the 
descriptions relating thereto) of the aforesaid patent application No. 01830562.3. 
For further details of the spatial noise estimation as described hereinabove, 
especially as expressed in equation (1), reference should also be made to US 
f'atent 6,108,455. 
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The spatial noise estimate NLn(x,y) as calculated in this manner is 
used for regulating the degree or strength of the filtering in the case in which the 
filtering of the pixel Pn(x.y) is exclusively of the spatial type. 

Once the spatial noise estimation phase of Snoise_est 23 has been 
5 completed, a subsequent inhomogeneity estimation phase Text_est associates an 
inhomogeneity index (or "texture" degree) TD(x,y) with the pixel to be filtered Pn{x,y) 
on the basis of a measure of the inhomogeneity (or, analogously, of the 
homogeneity) of the pixels forming part of the spatial working window SWn(x,y). 

The inhomogeneity index TD(x.y) serves to decide whether the pixel 
10 Pn(x,y) does or does not fonri part of a homogeneous region, this with a view to 
establishing whether or not the pixel in question (and therefore the corresponding 
spatial working window) will have to contribute to a spatio-temporal noise estimate 
to be described in greater detail further on. 

In fact, a homogeneous region can provide reliable information about 
15 the effective noise present in the image, because the fluctuations (/.a, the 
differences) between pixels forming part of a homogeneous region are 
substantially to be attributed to random noise. 

Persons skilled in the art are familiar with different metrics for 
calculating an inhomogeneity measure associated with a set of pixels, and for this 
20 reason we shall not here delve further into this matter. Among these metrics we 
shall here cite the following solely by way of example: maximum difference, 
minimum difference. MAD (Mean of the Absolute Differences), standard deviation. 
e)(traction of a distribution parameter from a histogram of the digital values of the 
pixels. 

25 In an embodiment that is particularly advantageous from a 

computational point of view, the parameter k„(x,y) used in the spatial noise 
estimation phase 23 (Snoise_est) is calculated on the basis of an 
inhomogeheity/homogeneity measure. More particulariy. it is obtained on the 
basis of a calculation of the differences between the pixel to be filtered and the 
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other pixels of the spatial working window. In that case the inhomogeneity 
estimation phase 24 (Text_est) can be Inserted in the spatial noise estimation 
phase 23 (Snoise_est) by calculating a single inhomogeneity measure that can be 
used both for estimating the spatial noise and for associating an inhomogeneity 
5 index TD(x,y) with the pixel to be filtered. 

A comparison phase 25 is used to verify whether the inhomogeneity 
index to(x,y) of the pixel Pn(x,y) is smaller than a predetermined threshold value Th 
{i.e., whether the pixel forms part of a region deemed to be homogeneous). 
Namely, when the inhomogeneity index To is smaller than a threshold Th, there is a 
10 local noise estimation phase 26 before the motion detection phase 27, but if To is 
larger than Th, there is a motion detection phase 27 immediately after the 
comparison phase. 

If this is not the case, the next step is the motion detection phase 27 
(Mot_det). But if the pixel forms part of homogeneous region, the motion detection 
15 phase 27 is preceded by a second local noise estimation phase 26 

(L_STnoise_est), i.e., performed on the pixel to be filtered, that serves to obtain a 
global noise estimate {i.e., for the entire image that is being processed). 

The local noise estimation phase 26 (L_STnoise_est) estimates a 
parameter - which may be statistical, for example - representative of the noise 
20 locally present in the spatial working window. This parameter Is calculated, for 
(example, as a local standard deviation of the spatial working window of the 
pixel Pn(x,y) or as some other analogous energy measure. 

Given the spatial working window SWn(x,y) of the pixel pn(x,y), the 
local standard deviation <t^J can be calculated in accordance with the following 
25 formula: 
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where N is the number of pixels forming part of the spatial working window 
SWn(x,y) (in this case N=9) and m is the mean of the digital values of these pixels. 

As already mentioned (and as is to be explained in greater detail 
further on), once the filtering of the image Imgn has been terminated, the various 
5 local standard deviations calculated in this manner for the pixels of the image 
Imgn that are deemed to form part of homogeneous regions will be used for 
updating a global noise estimate a^^^ , which we shall hereinafter refer to also as 
spatio-temporal noise estimate. In particular, this global estimate will be used for 
the spatio-temporal filtering of the subsequent image Imgn+i. 

10 The movement detection phase 27 (Mot_det) compares the pixels of 

the temporal working window TWn(x,y) with the pixels of the spatial working 
window SWn(x,y) in order to ascertain the presence of motion between the two 
working windows and possibly evaluate its magnitude. 

As is well known to a person skilled in the art, when using a "non- 

15 compensated motion" approach, appropriate precautions have to be taken to avoid 
the introduction during the spatio/temporal filtering of artifacts due to the motion 
between consecutive images. In particular, care must be taken to assure that the 
two working windows will not contain incongruent data on account of the motion 
between consecutive images or parts of them. 

20 For example, it may happen that one working window contains pixels 

that form part of an object, while the other window contains pixels that fomi part of 
the background, because the object has moved between one image and the next. 

Typical examples of artifacts that could be produced in these cases 
arie the presence of troublesome trails and so-called "ghost images" that become 

25 visible in the filtered image due to residual information of previous images. 

In a preferred embodiment, the motion detection phase (Mot_det) 
calculates as measure of motion a measure M(x,y) that is the sum of the absolute 
differences (SAD) between the pixels of the temporal working window and the 
pixels of the spatial working window. The greater the differences between the two 
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windows, the greater will be the value of this measure M(x,y), which can therefore 
be repreisentative of the motion between the two windows. 

The standard SAD measure as an isolated item is well known to 
persons skilled in the art and thus need not be described in detail herein. 

In a particularly advantageous embodiment variant when used with 
the present invention, the motion measure M(x,y) is a "modified" SAD. This 
measure is calculated by determining the difference in absolute value between the 
two working windows pixel by pixel, thus obtaining a working window difference 
DWn(x.y) given by: 

DWn(x.y)=ISWn(x.y) - TWn(x.y)| (3) 

Subsequently one proceeds to calculate the mean Wavg of the pixels 
of the operating window difference DWn(x,y). thus obtaining the modified SAD 
measure M(x,y)/ which is given by: 

M(x,y)=SAD(DWn(x,y)-Wavg). 

The modified SAD measure obtained in this manner is 
advantageous, because it makes it possible to avoid a change in lighting 
conditions being erroneously interpreted as a motion. 

With a view to avoiding an excessive sensitivity of the SAD value to 
the digital values of the pixels of the two working windows (which are contaminated 
by noise), another particularly advantageous embodiment variant makes it possible 
to introduce a slight quantization of the pixel values by reducing the accuracy of 
the pixel values from eight to seven bits when calculating the SAD. 

Another comparison phase 28 then checks whether the motion 
ftieiasure M(x,y) of the pixel Pn(x,y) is greater than a predetermined threshold value 
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When this is the case, the system concludes that there is excessive 
change between the two working windows and the subsequent filtering phase 29 
(S_filter) is therefore exclusively of the spatial type. 

The exclusively spatial filtering produces the filtered pixel f_Pn(x,y) 
5 from the pixels of the spatial working window. The strength of the filtering is 
regulated by the estimate of the spatial noise level NLn(x,y) calculated in noise 
estimation phase 23 (Snoise_est), In a preferred embodiment, the spatial digital 
filtering is carried out in accordance with the technique described in the previously 
mentioned European Patent Application No. 01830562.3, which obtains the filtered 
10 pixel as a weighted average (mean) of the pixels of the spatial working window 
(see, in particular, formula (9) of said application). 

Obviously, the exclusively spatial filtering is also carried out for all the 
pixels of the first image of the sequence, because temporal data are not yet 
available in this case. 

15 On the other hand, when the motion measure M(x,y) is smaller than 

the prisdetemiined threshold value Mh, the subsequent filtering phase 30 (ST_filter) 
is of the spatio/temporal type and is illustrated in greater detail in Figure 6. 

ST_filter 30 consists of a first filtering phase 33 (Duncan_Filt), which 
produces a provisional filtered pixel d jDn(x,y) in accordance with a filtering 

20 technique that is known by the name of Duncan filtering and will be described in 
greater detail further on. In this phase the provisional filtered pixel d j3n(x,y) is 
obtained from a subset of pixels forming part of both the working windows. 

As can be seen in Figure 6, when the detected motion is deemed to 
be sufficiently small, i.e., smaller than a further predetermined threshold value Mj 

25 that is smaller than the threshold value Mh, the provisional filtered pixel is not 
subjected to any further processing and one simply puts; 

LPn(x,y) = d j)n(x.y) (4) 
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In this case, therefore, the pixel Pn(x,y) is effectively filtered by means 
of a Duncan spatio/temporal filtering method. 

Vice versa, i.e.. in the case in which the detected motion is not 
negligible, the provisional pixel d_pn(x,y) is subjected to a further processing phase 
5 35 (Smooth_Filt), which produces the "definitive" filtered pixel f_Pn(x,y) in 
accordance with a smoothing operation as defined by the following formula: 

f.P„ix,y) = P„xd_ p„ ix,y) + il-p„)x p„(x,y) (5) 

where p„ is a multiplying factor comprised between 0 and 1 that may either 
depend on the motion measure M(x,y) or may be the same for all the images of the 

10 sequence. In a preferred embodiment, for example, /5„ is equal to about 0.75. 

Following the smoothing operation, the definitive filtered pixel is 
obtained from a portion (in this example 75%) of the value provided by the Duncan 
filtering and a portion (25%) of the value of the unfiltered pixel. In other words, the 
definitive filtered pixel is obtained from the sum of a fraction of the provisional 

15 filtered pixel and a fraction of the unfiltered pixel. 

This is done because, whenever there are non-negligible 
Incdhgruenoies due to motion between the working windows, It Is Important that 
onie should be able to "neglect" the temporal infomiation to a somewhat greater 
extent and attribute a little more Importance to the current image that is being 

20 fiiteroi. 

We shall now describe a particularly advantageous embodiment of 
the Duncan filtenng phase. 

Duncan filtering first selects a subset or "range" of pixels fomriing part 
of the two working windows by means of the so-called "Duncan Range Test" or 
25 "DRT" (in this connection see "Multiple range and multiple f-tests", D.B. Duncan, 
Biometrics, vol. 11., pp. 1-42, 1955). 

The application the DRT to digital filtering is known, for example, 
from European Patent Application EP 1 100 260 A1 by the present applicant. 



17 



The subsequent filtering operations of the pixel Pn(x,y) are then 
performed only on the pixels that form part of the range selected by means of the 
DRT. 

The purpose of the selection effected by means of the DRT is to 
5 exclude any pixels that, though forming part of the working windows, have had 
their value corrupted to an excessive extent by noise. 

For example, such pixels may be present due to the effect of a 
particular noise - known as "salt and pepper noise" - capable of bringing the 
digital values of some pixels up to the maximum value or down to the minimum 
1 0 value of the scale of the possible digital values. 

Selection by means of the DRT is also intended to exclude any pixels 
that are very different from the pixel to be filtered Pn(x,y). for example, on account 
of a different information content. One may think, for example, of the case in 
which the pixel to be filtered Pn(x,y) forms part of an "edge", while the working 
1 5 windows contain pixels that form part of the background of the scene. If the 

background pixels were not excluded, the image would suffer a considerable loss 
of definition as a result of the filtering. 

A selection made by nieans of the DRT has to identify a digital value 
interval SI (selection interval) having an appropriate width S such as to contain the 
20 largest possible number of pixels (in this case forming part of the working 

windows) similar to the pixel to be filtered Pn(x,y). The width S of the selection 
interval SI is correlated with the standard deviation of the noise to be filtered, which 
is assumed to be known. 

It is not necessary for this interval to be centered around the pixel to 
25 be filtered: if this were to be the case and if the pixel to be filtered were, for 
example, highly corrupted by noise, the test would exclude pixels useful for the 
filtering. 

In a preferred embodiment, the noise standard deviation used for the 
DRT selection during the filtering of the pixel Pn(x,y) of the image Imgn is the global 
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spatio/temporal noise estimate crf^ calculated in the filtering of the previous image 
lmgn-1. This choice is particularly advantageous from the point of view of 
computational resource optimization: in this way, in fact, one avoids having to carry 
out a complete scanning of the image Imgn that is to be filtered (an operation that 
5 serves only to estimate the noise) prior to the filtering in the proper sense of the 
term. 

The theory of DRT selection provides all the instruments needed for 
calculating the width S fronri the standard deviation of the noise and detemiining 
the optimal selection Interval SI: The Implementation of these Instruments, 
10 however, is very costly in computational terms and cannot readily be conciliated 
with the stringent requirements imposed by real-time image processing. 

In a preferred embodiment, an optimal compromise between 
reliability of the result and conriputational complexity is obtained by performing the 
selection of the pixel subset with one of the following three intervals SI1 , SI2, SI3, 
16 as shown in Figure 7, where: 

• the interval SI1 of width S is centered around the digital value 
DV of the pixel to be filtered Pn(x,y); 

• the interval Si2 of width S is centered around the digital value 
DV=p„(x.y)+af ; 

20 • the interval SI3 of width S is centered around the digital value 

DV=pn(x,y)-c7f . 

The interval to be chosen from among these three intervals 
S1 1 ,812,813 is the one that contains the largest number of pixels, which in Figure 7 
is the interval SI1. In this way good results are obtained even when the pixel to be 
25 filtered Pn(x,y) is a very noisy pixel. 

Still in a preferred embodiment, moi-eover, the width S of the 
selection interval SI is calculated as: 

S = 3>«rf ,g) 
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In a variant that is computationally costlier but yields optinnized 
performances, the choice of the selection interval is made by using appropriate 
weighting functions in accordance yN'Ah the method described in the 
aforementioned European Patent Application EP 1 100 260 A1 (where particular 
reference should be made to Figures 1b and 4). 

Once the pixels Pj most similar to the pixel to be filtered and 
contained in the two working windows and the selection interval SI have been 
identified by means of the DRT, the provisional filtered pixel d J3n(x,y) is calculated 
as the weighted mean of these pixels or, put in mathematical terms: 

d^pAx.y)=^X^jPj (7) 

where, preferably, the weighting coefficients aj are calculated as in the 
aforementioned European Patent Application EP 1 100 260 A1 (where particular 
reference should be made to page 6, lines 41-50). 

Coming back to Figure 3. the filtering - which, as previously 
explained, takes place in accordance with either phase 29 (S_filter) or phase 30 
(ST_filter) - is followed by a control phase 31 that checks whether the pixel Pn(x.y) 
that has just been filtered is the last pixel of the image Imgn. If Pn(x,y) is not the 
last pixel, the method represented as a succession of phases in Figure 3 is applied 
to the next pixel |n the scanning order, for example, the pixel pn(x,y+1). 

When it is the last pixel, on the other hand, there follows a global 
noise estimation phase of the spatio-temporal type based on the numerous local 
estimates calculated for the pixels of the image Imgn deemed to form part of 
homogeneous regions during the local noise estimation phase 26. In particular, 
these estimates are used to update a global noise estimate af^i of the spatio- 
temporal type that will subsequently be used in the spatio-temporal filtering of the 
next image Imgn+i. Preferaibly, the global estimate a^^^ should be representative 
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of the standard deviation of the noise and be calculated as the mean of the 
numerous local estimates (standard deviations) a^l . 

In some situations it may however happen that adjacent images have 
excessively discordant values of the global standard deviation af^, and the spatio- 
5 temporal filtering could therefore filter adjacent images with intensities that are too 
widely different. This would give rise to a bothersome flickering in the reproduction 
of the sequence. 

With a view to avoiding this drawback, a preferred embodiment 
modifies the global standard deviation a^^^ , originally calculated as the mean of 
10 the local standard deviations, by obtaining a time average in a recursive manner of 
a certain number (for example: two) of global standard deviations cr^^ relating to 
consecutive images. Put in mathematical terms, we thus have: 

<^f'i=y><cTl^-f(l-y)xaf (8) 

where y is a nurriber comprised between 0 and 1 , af^ is the global noise estimate 
15 as updated during the filtering of the previous image lmgn-i and used in the spatio- 
temporal filtering of the current image Imgn. For example, the number y may be 
eqiial to about 0.75. 

We shall now describe some embodiments alternative to the 
particular method described hereinabove by reference to Figure 3. 
20 As compared with Figure 3, the motion detection block 27 (Mot_det) 

of one of these embodiment variants limits itself to detecting the presence/absence 
of motion and as output provides a binary measure M(x,y) indicative of the 
presence/absence of motion. For example, the output may be M(x,y)=1 when the 
presence of motion is detected, othenvise the output will be M(x,y)=0. In this case 
25 the value of the threshold Mh may be chosen, for example, as equal to 0.5. 

Phase 24 (Mot_det) may detect the presence of motion by means of 
a "trail detection" method that operates by simply calculating the differences the 
isNO working windows pixel by pixel. If these differences all have the same sign, be 
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it positive or negative, the system detects the presence of motion, otherwise it 
detects the absence of motion. 

When the presence of motion is detected, the next step is the 
previously described spatial filtering phase 29. 
5 In the contrary case, i.e., when no motion is detected, this Is followed 

. by a spatio-temporal filtering phase 30 (ST filter) in which the filtered pixel is 
obtained by means of a Duncan filtering phase 33 (Duncan_Filt) - see Figure 6 - 
that may or may not be followed by a phase of smoothing filtering 35 
(Smooth_,Filt). 

10 In yet another embodiment variant, the two filtering phases 29 and 30 

(S_filter and ST_filter), which are, respectively, a spatial filtering and a spatio- 
temporal filtering, obtain the weighted means for calculating the filtered pixel 
LPn(x,y) by replacing some of the pixels that form part of the weighted mean by 
their respective filtered values whenever this value is already available in the 

15 output buffer. 

Experimental results have shown that the proposed filtering method 
is capable of providing concrete advantages in terms of both image quality and 
encoding/compression efficiency, and this notwithstanding the fact that it calls for 
the allocation of only modest computational and memory resources. This renders 
20 use of the method of the present invention particulariy advantageous in 
applications that call for real-time processing capacity. 

Referring to Figure 8, the curve denominated "CFA_filtered" 
reproduces, image by image, a quality measure for a sequence of three hundred 
imajges that were filtered in accordance with the present invention. The measure 
25 SNais obtained from filtered and interpolated images. 

The second curve in Figure 8, indicated by the denomination "Noisy", 
reproduces the same measure as obtained on the same sequence prior to filtering; 
in this case, once again, the measure was obtained from interpolated images. 
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The results of Figure 8 refer, in particular, to a measure known as 
PSNR (Peak-to-Peak Signal to Noise Ratio). The PSNR is a standard measure 
and is rejDresentative of the quality of an image; more particularly, it indicates the 
signal quantity present in an image as compared with the quantity of noise. 
5 The results of Figure 8 show that the filtered sequence is 

characterized by a higher PSNR measure (the gain is typically of the order of 3 
dB), which is indicative of a better quality. 

Figure 9 shows two processing schemes that can be used for 
obtaining a sequence of filtered images encoded/compressed in accordance with 

10 the MPEG4 standard from a sequence of noisy CFA images. 

More particularly, in the first of the two procedures, here 
denominated E1 , the CFA images are filtered by means of a method in accordance 
with the present invention by the filter 5 (CFA NF), after which they are interpolated 
by the block 7 (IGP) and encoded/compressed by the block 8 (MPEG4-Encoder). 

15 In the other procedure, here denominated E2, the noisy CFA images 

are first interpolated by the block 7 (IGP), after which - following a conventional 
filtering scheme they are filtered by means of a filtering method with motion 
compensation immediately prior to MPEG encoding/compression in Block 8. this 
method being indicated as MC_NF. In particular, the MC_NF method uses the 

20 motion estimate and the motion compensation of the MPEG encoder to perform a 
spatio-temporal digital filtering with motion compensation. 

Experimental results have shown that processing procedure E1 , /.e., 
the one in accordance with the present invention, makes it possible to obtain a 
gain of 20% in terms of bit rate with respect to the conventional procedure E2. 

25 This can be explained by considering the fact that processing 

procedure E2 estimates the motion and the mofion compensation from images that 
are affected by noise and cannot therefore furnish an optimal result, with 
consequent adverse effects as regards both image quality and compression 
efficiency. 

23 



The filtering method in accordance with the invention, which has 
been described hereinabove by reference to a preferred embodiment, can be 
implemented by utilizing hardware, software or a combination of hardware and 
software. In the latter case the method may be implemented in an application- 
5 specific integrated circuit (ASIC circuit). 

When it is implemented in a device for acquiring image sequences, 
the method in accordance with the present invention can be advantageously 
carried out by means of processing resources (DSP, for example) shared with 
other applications within the said device. 

10 Obviously, a person skilled in the art. especially when having to 

satisfy contingent and specific needs, could introduce numerous modifications and 
variants into the proposed method of filtering a digital image sequence, though 
without thereby overstepping the protection limits of the invention as defined by the 
claims set out hereinbelow. 

15 AH of the above U.S. patents. U.S. patent application publications, 

U.S. patent applications, foreign patents, foreign patent applications and non- 
patent publications referred to in this specification and/or listed in the Application 
Data Sheet, are Incorporated herein by reference, in their entirety. 

From the foregoing it will be appreciated that, although specific 

20 embodiments of the invention have been described herein for purposes of 

illustration, various modifications may be made without deviating from the spirit 
and scope of the invention. Accordingly, the invention is not limited except as by 
the appended claims. 
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CLAIMS 

1 . A method for filtering the noise of a sequence of digital images in 
video fonnat comprising the following phases: 

processing a first video image of the sequence to obtain a corresponding 
improved video image with reduced noise; 

processing at least one pixel of a second video image of the sequence 
that temporally follows said first image, said phase of processing at least one pixel 
providing a corresponding filtered pixel and including the following operations: 

selecting a first set of pixels comprising said at least one pixel and 
a plurality of pixels of the second video image spatially adjacent to it; 

selecting a second set of pixels comprising pixels of the improved 
video image homologous with the pixels of said first set; 

carrying out a digital filtering of a first type using pixels forming part 
of said first and second set to generate the filtered pixel. 

2. A method in accordance with Claim 1, comprising also a phase of: 
carrying out a first evaluation of motion of the at least one pixel, using 

pixels forming part of said first and second set; and in which said at least one pixel is 
such that said evaluation of motion is smaller than a first threshold value. 

3; A method in accordance with Claim 1 . comprising also a phase of: 

processing a further pixel of said second image, said phase of processing 
the further pixel providing a corresponding further filtered pixel and including the 
following operations: 

selecting a third set of pixels comprising said further pixel and a plurality of 
pixels of the second video image spatially adjacent to it; 

selecting a fourth set of pixels'comprising pixels of the improved video 
Image homologous with the pixels of said third set; 
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carrying out a further evaluation of motion of the further pixel, using pixels 
forming part of said third and fourth set; 

whenever said further evaluation of motion is smaller than said first 
threshold value, carrying out a digital filtering of a second type that generates the further 
filtered pixel by using exclusively pixels forming part of said third set. 

4. A method in accordance with Claim 1 , wherein each video iniage of 
the sequence is made up of a respective pixel matrix, the pixels of said matrix being 
associated on the basis of their respective positions with one of a set of chromatic 
components, and wherein said first and second set comprise pixels associated with the 
same chromatic component of the at least one pixel. 

5. A method in accordance with Claim 4, wherein the images of the 
sequence are in Bayer CFA format and said chromatic components form part of the set 
cbniiprising the color red, the color green and the color blue. 

6. A method in accordance with Claim 5, wherein the phase of 
selecting the first set of pixels is carried out by means of selection matrices that differ 
according to the chromatic component of the at least one pixel, the selection matrices 
being such as to select pixels that are situated in the neighborhood of the at least one 
pixel and having the same chromatic component as said at least one pixel, discarding 
the others, the selection matrices being also identical for the chromatic components red 
and blue. 

7. A method in accordance with Claim 1 that includes also a phase of 

GL 

making an estimate a statistical parameter " representative of the global noise 
present in said first image (Imgn-i), the digital filtering of the first type utilizing also said 
statistical parameter. 
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8. A method in accordance with Claim 7, comprising also the following 

phases: 

selecting a plurality of pixels of the first image; 

calculating a plurality of local estimates, calculating for each given pixel of 
said plurality of pixels a respective estimate of a statistical parameter representative of 
the local noise in present in a neighborhood of the given pixel; and wherein said 
estimate of the statistical global noise parameter a]^^ is obtained from said plurality of 
local estimates. 

9. A method in accordance with Claim 8, wherein said local estimates 
are local variance measures. 

10. A method in accordance with Claim 8, wherein said plurality of 
pixels conriprises pixels forming part of homogeneous regions of the first image. 

1 1 . A method in accordance with Claim 1 , wherein said digital filtering 
of the first type utilizes a subset of pixels forming part of said first and said second set of 
pixels, said subset being identified by means of a further selection phase carried out in 
adcordance with a Duncan Range Test. 

12. A method in accordance with Claim 3, comprising also a phase of 
estimating for the further pixel a further statistical parameter representative of the noise 
present on the pixels of said third set, said further parameter estimated also on the 
bksis of the specific color of the further pixel, the digital filtering of the second type 
utilizing said further parameter. 

13. A filter for reducing the noise of a sequence of images in CFA 
format, characterized in that said noise reduction is obtained by means of a method in 
accordance with claim 1. 



27 



14. An acquisition device for acquiring a sequence of digital images in 
CFA format, comprising a sensor for acquiring said images, said sensor comprising a 
CFA filter, characterized in that the sequence of images in CFA format is processed by 
rheans of a noise filtering method in accordance with claim 1 . 

15. A method of filtering noise from a digital video image comprising: 
processing a first image to generate an improved image; 
processing a second, subsequent image; 

selecting a first pixel from the second image during the processing of the 
second image; 

selecfing a first set of pixels in the second image that are a predetermined 
spatial relationship to the first pixel; 

locating a corresponding second set of pixels in the first image that 
correspond to the first set of pixels in the second image; and 

filtering the first pixel using data from both the first set and the second set 

of pixels. 

16. The method according to claim 15 further including: 
performing spatial filtering on the first pixel; 

determining a motion component between the first set of pixels and the 
second set of pixels; and 

carrying out a motion compensator filtering if the first motion component 
between the first set and the second set is above a selected threshold and not carrying 
out motion compensation filtering if the motion component between the first set and the 
second set is below a selected threshold 
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ABSTRACT OF THE DISCLOSURE 

A method for filtering the noise of a sequence of digital images in CFA 
format comprising the following phases: 

processing a first CFA image of the sequence to obtain a corresponding 
improved CFA image with reduced noise; 

processing at least one pixel of a second CFA image of the sequence that 
temporally follows said first image, said phase of processing at least one pixel providing 
a corresponding filtered pixel and including the following operations: 

selecting a first set of pixels comprising said at least one pixel and 
a plurality of pixels of the second CFA image spatially adjacent to it; 

selecting a second set of pixels comprising pixels of the improved 
CFA image homologous with the pixels of said first set; 

carrying out a digital filtering of a first type using pixels forming part 
of said first and second set to generate the filtered pixel. 

851 063.485/ 404798 J 
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